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ABSTRACT 

Training  is  increasingly  becoming  technology-based,  swapping  classroom  time  and  live  instmctors  for  distance 
learning,  serious  games,  and  simulation  exercises.  This  presents  both  challenges  and  opportunities  for  tailoring 
training  to  accommodate  differences  in  trainees’  backgrounds,  prior  knowledge,  and  abilities.  Our  interest  is  in 
building  comprehensive  technology-based  instructional  environments  that  adapt  to  differences  and  changes  in 
cognitive  factors — experience,  knowledge,  skills,  and  attitudes. 

We  describe  a  space  of  techniques  for  adapting  instruction  in  temis  of  (1)  the  aspects  of  a  student  model  that  inform 
individualized  instructional  decision-making,  and  (2)  the  aspects  of  the  instructional  experience  that  can  be  adapted 
based  on  those  factors.  Student  model  elements  include:  (la)  information  on  experience  extracted  from  background 
questionnaires,  (lb)  records  of  student  exposure  to  instruction  and  exercises,  (lc)  assessments  of  student 
performance  during  exercises,  and  (Id)  overall  student  mastery  estimates  for  system  learning  objectives. 
Instructional  adaptations  include:  (2a)  choices  of  didactic  instruction,  (2b)  choices  of  exercise  scenarios,  (2c)  choices 
affecting  selection  and  delivery  of  exercise  performance  hints  and  feedback,  and  (2d)  choices  controlling 
pedagogically  significant  behaviors  of  simulated  agents  within  scenarios. 

We  give  examples  from  a  problem-based  learning  environment  intended  to  train  U.S.  Army  Battle  Captains  on  how  to 
supervise  current  operations  in  battalion  Tactical  Operations  Centers  (TOCs).  A  prototype  implementation  provides 
a  unified  environment  combining  instmctional  presentations,  a  scenario -driven  TOC  simulation,  and  machinery  for 
controlling  simulation  behavior,  student  assessment,  and  instmctional  interventions.  We  describe  the  student 
modeling  and  instmctional  control  components,  emphasizing  the  breadth  of  instmctional  adaptation  supported.  We 
highlight  how  a  control  mle  language,  in  the  context  of  the  overall  system,  will  enable  experimentation  with  alternate 
adaptation  strategies.  Such  an  environment  is  an  essential  tool  for  establishing  an  empirical  basis  for  guiding  future 
deployment  of  adaptive  instmctional  systems. 
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WHY  SHOULD  TRAINING  BE  ADAPTIVE? 

A  MOTIVATING  EXAMPLE 

Empirical  results  support  common  sense  when  it  comes 
to  instruction:  students  do  better  when  given 
individualized  attention  (Bloom,  1984).  Mass 
instruction — in  classrooms  and  lecture  halls — is 
primarily  an  economic  compromise.  Though  there  are 
benefits  to  interaction  and  collaboration  at  the  level  of 
seminars  and  small  work  groups,  nobody  really  thinks 
that  every  student  needs  to  hear  exactly  the  same 
information  in  the  same  sequence  and  format.  We  know 
that  some  students  are  wasting  time  by  going  too 
slowly,  and  others  need  still  more  time  and  attention  to 
come  up  to  criterion.  Ideally,  computer-based  training 
environments  should  adapt  to  the  needs  of  each 
individual  student,  just  as  a  good  instructor  might,  were 
the  instructor  able  to  devote  their  full  attention  to  that 
one  student. 

The  need  for  adaptive  instruction  is  increased  in 
training  applications  where  the  variation  in  student 
background  and/or  ability  is  large,  or  where  the  costs  of 
wasting  student  time  are  high.  We  have  been  focusing 
on  one  such  Army  application  that  has  the  additional 
property  of  being  only  weakly  supported  by  existing 
Army  training.  The  problem  we  have  been  addressing 
is  development  of  computer-based  adaptive  training  for 
battalion  battle  captains  (BCs). 

hi  the  Army,  a  battalion-level  battle  captain  is 
responsible  for  overseeing  the  information  flow  and 
tracking  battles  and  other  operations  in  the  Tactical 
Operations  Center  (TOC)  (de  Oliveria,  1995;  Wampler,  et 
al.,  1998).  Other  personnel  are  primarily  responsible  for 
monitoring  and  logging  the  various  information 
channels,  such  as  the  radio  nets,  e-mail,  chat,  and 
battlespace  mapping  and  unit  tracking  software,  but  the 
battle  captain  is  responsible  for  noticing  whenever 
events  have  moved  beyond  the  routine  execution  of 
plans  and  will  now  require  actions  and  decisions.  The 


battle  captain  ranks  below  the  S3  (operations  officer), 
the  XO  (executive  officer),  and  the  battalion  commander 
in  the  chain  of  command,  but  the  senior  officers  are 
frequently  away  from  the  TOC,  leaving  the  battle 
captain  in  charge  of  implementing  the  planned 
operations  and  making  decisions  to  support  those 
operations  when  needed. 

The  BC  position  is  not  defined  by  doctrine,  yet  a  BC 
can  be  found  in  nearly  every  battalion  or  brigade 
tactical  operations  center  (TOC).  Staff  jobs  in  general, 
and  the  BC  job  in  particular,  are  often  considered 
undesirable,  relative  to  commands  and  billets  that  take  a 
soldier  into  the  field.  Officers  and  enlisted  men  with  a 
broad  range  of  backgrounds  may  find  themselves 
slotted  into  the  BC  post  at  short  notice.  At  the 
battalion  level,  a  BC  night  be  a  captain,  lieutenant,  or 
even  an  NCO;  brigade  BCs  are  often  majors.  The  BC 
job  also  varies  widely  with  the  leadership  style  of  the 
commander,  XO,  or  S3.  In  our  analysis  of  the  domain, 
we  focused  on  five  terminal  learning  objectives  for 
battle  captains,  summarized  in  Table  1. 


Table  1.  Battle  Captain  Learning  Objectives 


Learning  Objective 

Description 

Help  the  Commander 
Manage  the  Force 

Understand  the  mission 
and  battalion  capabilities; 
issue  warnings  and  orders 

Maintain  Situation 
Awareness 

Keep  track  of  actors, 
states,  and  events  along 
with  their  implications 

Manage  Information 

Assemble,  assess,  filter, 
pull,  and  push  information; 
use  appropriate  C3  system 

Support  Decision 
Making  and  Action 

Recognize  decision  points; 
who  needs  to  decide  and 
by  when? 

Help  Manage 
the  Fight 

Anticipate,  prioritize, 
reallocate,  and  orchestrate 
resources  and  actions 
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Our  challenge  then  is  to  develop  an  adaptive  battle 
captain  training  system  that  (1)  teaches  relatively  high 
level  situation  assessment  and  decision-making  skills, 

(2)  can  cope  with  variation  in  student  background,  and 

(3)  can  be  extended  to  also  address  variation  in  the 
specifics  of  target  behaviors. 

APPROACHES  TO  ADAPTATION 

Most  training  systems  offer  students  some  mixture  of 
didactic  instructional  presentations,  practice  problems 
or  exercises,  and  assessment  instruments.  In  many 
cases,  the  practice  problems  do  double  duty,  serving  as 
interim  assessments.  Adaptation,  in  such  systems 
involves  taking  the  individual  student's  needs  into 
account,  in  order  to  adjust  the  choice,  mix,  sequencing, 
and/or  composition  of  instruction,  practice,  and 
assessment  elements.  The  system’s  model  of  the 
individual  student  and  their  needs  is  built  up  from 
ongoing  assessment,  possibly  primed  by  some  initial 
demographic  data. 

Simple  Adaptation  for  Simple  Skills 

In  training  applications  that  emphasize  concrete  and 
atomic  learning  objectives  it  is  often  possible  to 
challenge  a  student  with  a  great  number  of  short, 
focused  problems  or  quiz  items,  each  of  which 
addresses  one  (or  perhaps  a  small  number  of)  learning 
objectives  with  little  ambiguity.  Such  a  system  can 
adapt  to  individual  students  simply  by  choosing  fewer 
or  more  problems  for  each  objective  as  a  student 
demonstrates  greater  or  lesser  competence  on 
objectives,  by  giving  correct  or  incorrect  answers  to 
challenge  problems . 

Similarly,  the  system  may  have  a  repertoire  of  didactic 
instruction  (e.g.  texts,  illustrations,  or  multimedia 
presentations)  intended  to  convey  key  knowledge  that 
should  help  students  succeed  on  practice  and 
assessment  problems.  The  presentations  are  typically 
used  to  prepare  students  for  problems,  remediate  gaps 
in  their  knowledge  identified  by  problem  failures,  or 
reinforce  key  points  by  repetition  and/or  variation  in 
presentation.  The  system  will  have  occasion  to  offer  a 
student  fewer  or  more  of  these  instmctional 
presentations  depending  on  how  long  it  takes  them  to 
succeed  on  enough  problems  to  be  considered 
competent  at  the  covered  objectives. 

This  kind  of  approach  is  likely  to  work  best  for  training 
isolated  sub-skills — often  components  oflarger desired 
behaviors.  As  an  example,  in  order  to  effectively  track  a 


battle,  a  battle  captain  must  be  able  to  interpret  map 
symbology  and  unit  icons.  There  is  a  lot  to  leam  here 
(i.e.  much  of  FM  101-5-1  Operational  Terms  and 
Graphics),  but  as  an  isolated  skill,  it  is  relatively  easy  to 
tell  whether  a  student  can  identify  or  use  any  particular 
graphic;  a  system  can  easily  adapt  by  choosing 
targeted  instruction  and  problems  or  quiz  questions. 

Complex  Skills  Training 

Map  interpretation  in  support  of  battle  tracking  is, 
however,  just  one  small  part  of  a  battle  captain’s  job. 
The  real  core  of  the  job  is  maintaining  situation 
awareness — for  what  is  going  on  both  in  the  field  and 
in  the  TOC — ensuring  information  is  flowing  to  all  the 
right  places,  recognizing  what  must  be  done  to  keep 
things  on  track,  and  knowing  how  to  respond  when 
they  go  off  the  rails.  Furthermore,  a  battle  captain 
cannot  just  be  reactive;  he  must  anticipate  what  night 
go  right  or  wrong,  and  how  he  can  help  the  unit  exploit 
success  or  adjust  to  challenges. 

We  are  interested  in  training  these  kinds  of  higher  level 
cognitive  skills.  The  training  challenge  is  that  such 
skills  can  only  really  be  exercised  in  complex  contexts : 
the  entire  state  of  the  battalion  (setting,  mission, 
commander’s  guidance,  current  activities,  etc.),  the 
availability  of  resources  at  various  levels  (platoons 
already  on  patrol,  prior  commitments  of  quick  response 
or  surveillance  resources,  availability  of  brigade  or  even 
division  assets,  etc.),  and  the  larger  situation  (recent 
history,  current  intelligence,  host  nation  and  insurgent 
activities,  etc.)  can  all  affect  the  battle  captain’s 
assessments,  decisions,  and  actions.  The  battle  captain 
does  not  work  alone,  but  in  concert  with  a  host  of  other 
actors ;  coordination  is  carried  out  through  a  wide  range 
of  communications  channels . 

When  the  dominant  aspect  of  the  skill  is  recognizing 
and  analyzing  complex  contexts  that  demand  different 
decisions  and  actions,  it  is  hard  to  frame  small  isolated 
problems  that  support  valid  practice  and  assessment. 
For  such  skills,  problems  and  quiz  questions  are  best 
replaced  with  scenarios  and  simulation-based  exercises. 
Instead  of  focusing  on  a  single  clear  learning  objective, 
such  exercises  tend  to  address  an  interrelated  set  of 
objectives.  This  reflects  (1)  a  desire  to  leverage  the  time 
spent  establishing  the  context  to  accomplish  as  much 
training  as  possible,  (2)  the  reality  that  large-scale 
contextualized  skills  typically  flow  in  natural  (often 
somewhat  variable)  sequences,  and  (3)  the  constraint 
that  complex  decisions  often  depend  on  interaction 
among  multiple  goals  and  activities.  For  example,  if  one 
of  a  battalion’s  patrols  gets  stuck  out  in  the  field,  there 
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are  a  number  of  issues  that  might  be  worth  considering: 
Will  they  be  able  to  get  themselves  unstuck,  or  will  they 
need  recovery  assets  to  assist  them?  Will  they  (or 
newly  tasked  recovery  assets)  be  subject  to  actual  or 
potential  threats  that  require  additional  surveillance  and 
security  assets?  How  does  their  immobilization  (and 
the  assignment  of  other  assets  to  assist  them)  affect  the 
prior  battalion  plan  and  allocation  of  forces  throughout 
the  sector? 

With  respect  to  the  learning  objectives  summarized  in 
Table  l,  this  one  simple  situation  can  provide  a  context 
for  exercising  skills  undereach  of  the  major  headings: 

1.  Help  the  Commander  Manage  the  Force:  The 

BC  must  understand  the  capabilities  of  the 
various  units  and  resources  at  the  battalion’s 
disposal.  The  BC  must  issue  timely  “warning” 
orders,  and  appropriate  “fragmentary”  orders 
to  those  resources.  [Is  it  likely  that  the 
stranded  unit  can  self-recover  its  immobilized 
vehicle?  What  kind  of  recovery  asset  would 
be  appropriate  to  recover  the  kind  of  vehicle 
that  is  immobilized?  What  are  the  organic 
surveillance  and  security  capabilities  of  the 
stranded  patrol  and  of  a  candidate  recovery 
unit?  How  long  should  we  expect  another 
unit  to  take  in  reaching  the  stranded  unit? ] 

2.  Maintain  Situation  Awareness:  The  BC  must 

track  battalion  forces  in  sector — their 

locations,  missions,  and  status.  The  BC’s 
model  must  include  an  understanding  of  who 
knows  what  about  current  events.  [Where  is 
the  stranded  unit  and  what  were  they 
responsible  for  doing?  What  other  units  are 
in  sector  that  might  be  able  to  assist  them? 
What  other  units  are  available  to  cover  for 
any  gaps  in  combat  power? ] 

3.  Information  Management:  The  BC  must  be 
able  to  use  all  the  various  communications 
system  effectively,  and  must  pull  and  push 
information  as  needed  to  ensure  that  he 
maintains  a  good  picture  of  what  is  happening, 
and  that  other  key  decision-makers  and  actors 
remain  informed.  [Is  there  additional 
information  needed  to  fully  understand  the 
situation  ( the  kind  of  vehicle,  the  security 
situation)?  Are  there  others  who  need  to 
know  about  the  situation  ( the  maintenance 
company,  the  quick  reaction  force,  senior 
leaders)?  What  are  the  best  mechanisms  for 
pulling  or  pushing  needed  information?  ] 


4.  Decision  Making  and  Action:  The  BC  must 
understand  when  particular  kinds  of  decision 
must  be  made.  He  must  know  which  decisions 
he  is  empowered  to  make,  and  when  he  must 
get  others  in  the  loop.  [Is  there  a  significant 
decision  to  be  made  here?  If  so,  who  can 
niake  it,  and  when  does  it  have  to  be 
decided ?] 

5.  Help  Manage  the  Fight:  The  BC  must  notice 
and/or  anticipate  changes  in  situation  and 
deviations  from  plan,  including  threats  and 
opportunities.  The  BC  must  help  manage 
resources  to  address  such  situations,  noticing 
when  new  or  different  resources  are  needed, 
reallocating  resources  under  battalion  control 
and  requesting  additional  resources  as 
needed.  The  BC  must  sequence  and  prioritize 
TOC  activities,  which  can  include  launching 
and  running  pre -defined  battle  drills  that 
orchestrate  response  to  common  occurrences. 
[Given  this  situation  (not  yet  especially 
serious)  is  there  a  standard  response  to  set  in 
motion?  What  else  might  happen  that  could 
niake  it  more  serious?  What  kind  of  resources 
might  be  needed  in  those  cases?  What  can  or 
should  be  done  to  prepare?] 

This  quite  simple  example  shows  that  if  we  want  to  train 
BCs  on  cognitive  skills  that  constitute  the  heart  of  their 
job,  then  we  naturally  end  up  with  exercises  that 
address  many  learning  objectives  at  once.  Once  we  set 
up  a  complex  situation  with  many  issues  at  stake,  it  is 
natural — both  realistic  and  efficient — to  draw  the 
situation  out — to  provide  time  for  information  gathering 
and  decision-making,  which  is  also  time  during  which 
the  situation  may  change  in  ways  that  raise  still  more 
potential  training  issues. 

For  instance,  if  we  give  the  student  time  to  decide  what 
(if  anything)  to  do  about  an  immobilized  patrol,  then  (as 
often  happens  in  the  real  world)  that  patrol  may  become 
a  target  of  insurgent  attack.  Or  conversely,  if  we  were 
trying  to  set  up  an  exercise  about  a  patrol  coming  under 
insurgent  attack,  it  might  be  useful  to  have  a  stream  of 
prior  events  and  message  traffic  that  affect  the  way  that 
attack  should  be  thought  about  (e.g.  how  would  the 
situation  differ  if  we  knew  the  attacked  patrol  was 
immobilized,  and  perhaps  that  we  had  already 
dispatched  supporting  resources  to  aid  them?). 
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Adaptation  for  Complex  Skills  Training 

We  have  focused  on  the  issue  of  training  for  complex 
cognitive  skills — contextually  sensitive  interpretation 
and  decision  making  skills.  We  have  suggested  that 
such  training  benefits  from  complex  exercises — 
extended  simulation-based  decision  making  and 
rationale  exploration  interactions  that  address  multiple 
learning  objectives  in  the  context  of  multiple 
performance  goals.  The  question  then  is  what  can  or 
should  adaptation  look  like  in  such  applications? 

The  dominant  issue  is  affecting  adaptation  here  is  that 
the  grain  size  of  exercises  is  quite  large.  Budding  a 
training  system’s  practice  component  around  extended 
simulations  versus  stand-alone  problems  raises  a  set  of 
sub-issues:  (1)  it  takes  longer  for  a  student  to  engage  in 
each  exercise,  so  it  is  even  more  essential  to  adapt 
exercise  content  to  individual  student  needs;  (2)  it  takes 
longer  to  create  or  author  such  large  exercises  so 
developing  a  substantial  repertoire  of  exercises  to 
choose  among  may  become  expensive;  (3)  each  exercise 
may  naturally  addresses  a  wide  swath  of  learning 
objectives  so  matching  exercise  to  student  learning 
needs  may  be  more  difficult. 

The  natural  response  to  all  of  these  problems  is  to 
address  the  root  cause,  and  to  see  if  we  can 
disaggregate  large-scale  simulation  exercises  into 
smaller  components  that  can  be  composed  to  better  suit 
individual  student  needs.  We  have  identified  two  main 
approaches: 

1.  Storyline  Composition:  Prepare  a  number  of 
parameterized  partial  scenarios,  each  of  which 
represents  a  coherent  extended  thread  of 
activity,  and  which  are  designed  to  be  woven 
together  in  various  ways  to  create  a  complete 
scenario.  This  approach  ought  to  enable 
variation  of  learning  objectives  covered  (a 
composed  scenario  would  typically  address 
the  union  of  the  learning  objectives  covered 
by  the  individual  storylines).  It  could  also 
enable  variation  in  difficulty  (by  manipulating 
the  number  and  complexity  of  storylines 
integrated,  and  the  time -phasing  of  the 
demands  they  place  on  the  student). 

2.  Scenario  plus  Injects:  Design  a  scenario  with 
a  relatively  stable  backbone  of  events  and 
issues,  and  then  identify  points  where 
additional  learning  opportunities  can  be 
injected  by  introducing  some  variant  behavior 
and  tracing  out  its  consequences.  The  same 


basic  concept  can  be  used  to  identify 
“rejects” — pieces  of  a  scenario  that  can 
optionally  be  left  out  to  remove  challenges 
what  seems  appropriate  to  a  particular  student. 
The  concept  can  also  be  extended  to  the  case 
where  injected  behaviors  have  the  effect  of 
simplifying  or  removing  challenges  (e.g.  if  a 
character  takes  responsibility  for  a  task  the 
student  would  otherwise  have  to  do). 

In  addition  to  adapting  the  practice  component  of 
training  (here  extended  simulation  exercises),  it  is  also 
worth  reviewing  options  for  adapting  the  instmctional 
component.  The  simple  objective -centric  approach  to 
instruction  is  likely  to  provide  a  workable  starting  point. 
To  the  extent  that  the  learning  objectives  are  somewhat 
independent,  the  system  can  at  any  moment  identify 
some  set  of  objectives  that  are  relevant  (e.g.  needed  to 
succeed  at  an  upcoming  challenge,  or  requiring 
remediation  based  on  performance  in  a  recent  exercise). 
A  range  of  instructional  presentations  are  prepared  for 
all  the  system’s  learning  objectives  and  they  are 
deployed  as  needed. 

In  the  kind  of  training  application  we  are  concerned 
with  here,  there  is  room  for  improvement  on  this  basic 
approach. 

1.  Integrating  Multiple  Presentations:  Since 
there  are  likely  to  be  multiple  objectives  in  play 
at  any  given  time,  there  may  be  several  (or 
many)  objective-specific  instructional 
presentations  nominated  or  queued  up  for  the 
student.  Sequencing,  bridging,  interleaving 
and/or  condensing  those  presentations  to 
form  a  coherent,  effective,  and  efficient  overall 
presentation  would  be  a  major  challenge.  For 
the  most  part,  we  do  not  intend  to  address  this 
issue  since  we  do  not  propose  to  generate 
novel  media  on  the  fly.  The  exception  is  that 
we  can  optionally  control  sequencing  based 
on  an  understanding  of  dependencies  among 
learning  objectives  (e.g.  prerequisites, 
subordinates,  etc.).  However,  in  practice  the 
sequencing  is  often  left  to  the  student  who 
can  choose  among  a  number  of  relevant 
presentations  identified  by  the  system. 

2.  Interaction  Presentations:  More  interesting 
than  sequencing,  smoothing,  or  merging  sets 
of  presentations  is  the  case  where  the  best 
instruction  addresses  some  interaction  among 
objectives.  For  instance,  the  BC  must 
understand  capabilities,  responsibilities,  and 
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availabilities  of  key  assets,  and  juggle  their 
allocation  across  actual  and  potential  tasks;  a 
learning  objective  about  anticipating  resource 
needs  may  interact  with  an  objective  about 
appreciating  the  capabilities  of  some  resource 
(e.g.  some  air  assets  can  provide  surveillance 
support,  while  others — with  more  limited 
availability — can  provide  both  surveillance 
and  security).  Introducing  instruction  that 
explicitly  deals  with  interactions  among 
objectives  expands  the  space  of  possible 
presentations.  We  continue  to  assume  that 
system  instmction  is  primarily  pre-packaged 
media  presentations.  In  this  case,  assuming 
that  a  given  interaction  has  been  recognized  as 
significant  enough  to  warrant  preparation  of 
focused  instruction,  the  remaining  issue  is  to 
allow  indexing  of  instruction  on  multiple 
objectives  and  to  prefer  presentations  that 
address  multiple  objectives  currently  active  for 
a  student. 

3.  Dynamic  Instruction:  In  some  cases  we  may 
wish  to  relax  the  assumption  that  instmction  is 
delivered  as  pre-packaged  media.  For 
instance,  instmction  that  describes  common 
causal  patterns  in  the  domain,  or  appropriate 
action  sequences,  may  benefit  fromdiscussion 
of  examples.  If  the  student  has  just  played 
through  a  scenario  that  illustrates  the  causal 
chain,  or  that  called  for  a  given  actions 
sequence,  then  it  makes  sense  to  discuss  the 
general  point  in  the  context  of  that  recent 
example.  This  kind  of  instmction  blurs  the  line 
with  scenario  content,  and  in  fact  is  probably 
best  authored  as  part  of  a  given  scenario, 
making  use  of  the  same  behavior  encoding 
techniques  (mles  and  scripts)  that  are  used  to 
generate  other  character  and  embedded  tutor 
behaviors. 


PROBLEM  BASED  LEARNING  (PBL) 

AND  MULTI-FORMAT  SIMULATION 

The  general  training  approach  outlined  in  this  paper  is  a 
form  of  problem-based  learning  (PBL,  e.g.,  Savery  & 
Duffy,  1995)  adapted  for  computer-supported 
individual — versus  team  or  group — instmction.  We 
combine  computer  simulation  with  automated 


assessment  and  embedded  coaching.  The  simulation 
allows  students  to  “sense”  and  “act”  in  a  world  that 
gives  them  relevant  cues  and  responses.  The 
assessment  builds  up  the  individualized  student  model 
that  ultimately  enables  adaptation.  The  coaching  helps 
students  leam  in  context,  scaffolds  them  to  succeed  at 
challenges  that  might  otherwise  be  beyond  their  current 
abilities,  and  helps  keep  them  within  the  envelope  of 
useful  training  and  effective  simulation. 

PBL  is  a  constmctivist  approach  to  training  that 
contrasts  with  more  common  direct  instmction 
approaches.  While  direct  instmction  involves  “telling” 
trainees  what  to  do,  constmctivist  approaches 
emphasize  the  trainee’s  role  in  developing  (or 
“constructing”)  his/her  own  knowledge  through 
discovery,  inquiry,  or  exploration  (Duffy  &  Jonassen, 
1992).  Constmctivist  approaches  such  as  PBL  have 
become  more  widespread  in  the  military  in  recent  years, 
for  example  through  initiatives  such  as  the  Adaptive 
Leaders  Methodology  (ALM,  e.g,  Vandergriff,  2006) 
and  the  Combat  Application  Training  Course  (CATC). 

In  order  to  deal  with  the  open -textured  reality  of  a 
complex  interpretation  and  decision  making  domain,  we 
advocate  relying  on  a  layered  mixture  of  mles  and 
scripting  to  capture  non -player  character  behaviors,  as 
well  as  automated  tutor  assessments  and  responses. 
Rules  can  be  efficient  ways  to  encode  reusable 
behaviors,  when  they  apply.  However,  in  the  real 
world,  every  mle  has  its  exceptions.  Sometimes  those 
exceptions  can  be  codified  as  additional  mles  that  apply 
in  more  narrow  circumstances.  Sometimes  all  we  can  do 
is  say  that  in  a  particular  circumstance — for  instance  a 
specific  training  scenario — that  a  particular  specific 
behavior  is  called  for.  Thus  we  may  have  general  mles, 
more  specific  mles,  and  scripted  responses,  depending 
on  the  behavior  and  the  situation. 


AN  EXAMPLE  SYSTEM 

To  illustrate  many  of  the  points  above,  we  describe  the 
intelligent  tutoring  system  (ITS)  we  have  been 
developing  for  battle  captain  (BC)  adaptive  training. 
Figure  1  shows  the  main  simulation  screen  of  the  BC 
ITS  as  it  appeared  during  a  recent  experiment  with 
Soldiers . 
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Figure  1.  Sample  Screen  from  Battle  Captain  Training  System. 


The  left  margin  contains  simulation  controls  (e.g.  time 
manipulation,  and  access  to  the  embedded  tutor),  as 
well  as  buttons  that  open  “Wingboards”  containing 
reference  materials  of  the  kind  that  would  be  available  in 
a  typical  TOC.  In  the  main  area,  the  top  section  offers 
controls  simulating  a  variety  of  communications 
channels,  including  (1)  face-to-face,  (2)  radio,  (3) 
telephone,  and  (4)  digital  text;  the  bottom  section  offers 
a  situation  map  that  offers  some  of  the  features  of 
FBCB2,  including  blue-force  tracking.  The  screen 
layout  reflects  the  fact  that  we  are  emphasizing  battle 
tracking,  and  that  situation  assessment  and  distributed 
team  coordination  are  major  parts  of  the  BCjob. 

The  centrality  of  communication  with  simulated  agents 
motivates  one  of  the  major  research  thrusts  of  this 
work:  development  of  spoken  and  written  natural 
language  dialog  support.  The  three  columns  for  face- 
to-face,  radio,  and  telephone  communications  provide 
“push-to-talk”  buttons  tied  to  a  speech  input 


processor;  character  responses  are  fed  to  a  speech 
synthesis  system.  The  digital  text  pane  is  likewise  tied 
to  a  text  input  processor.  Natural  language  processing 
is  a  hard  problem,  especially  in  relatively  open-ended 
domains,  such  as  this  one,  and  so  the  system’s 
language  interpretation  capabilities  are  experimental.  A 
menu  system  provides  an  alternative  input  format. 

The  student  is  introduced  to  the  simulation  mechanics 
(“buttonology”)  and  the  simulated  world  (“setting”) 
through  a  series  of  narrated  “micro -scenarios”  that  ask 
them  to  try  out  various  system  controls  and  study  the 
range  of  wingboards.  A  planned  extension  will  also 
introduce  them  to  their  simulated  unit’s  SOPs  and  battle 
drills  to  provide  further  practice  and  orientation.  A 
complex  simulation  exercise  starts  with  a  narrated 
briefing  on  the  situation  and  planned  operations. 
Following  the  briefing,  time  may  skip  forward  to  a  point 
where  significant  events  start  to  happen,  as  reflected  in 
communications  and/or  map  updates . 
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The  student  is  not  left  to  cope  with  the  complexity  of 
the  simulated  world  on  their  own.  An  automated  Coach 
is  available  to  provide  hints  and  prompts  to  guide 
action,  as  well  as  positive  and  negative  feedback  to 
highlight  successes  and  failures  (especially  when  the 
natural  consequences  of  student  actions  would  be  too 
long  delayed,  or  too  attenuated  to  serve  as  effective 
feedback).  Figure  2  shows  the  coaching  window,  with  a 
sequence  of  successively  more  directive  suggestions 
for  the  student. 


Figure  2.  Automated  Coaching  Window. 

The  introduction  of  automated  coaching  suggests  yet 
another  way  in  which  a  training  system  can  adapt  to 
individual  student  needs.  If  we  think  of  the  coach  as  a 
non-player  character,  then  the  choice  to  deliver  or 
suppress  coaching  seems  similar  to  the  “Scenario  plus 
Injects”  framing  of  adaptation;  however  coach  actions 
are  simpler  since  they  do  not  have  to  make  sense  with 
respect  to  a  character’s  role,  and  they  do  not  modify  the 
further  evolution  of  the  scenario.  Alternately,  we  can 
frame  coaching  as  “Dynamic  Instruction;”  but,  again 
the  coaching  we  have  so  far  developed  consists  of 
relatively  isolated  comments  rather  than  extended 
dialogs  exploring  causality  and  rationale. 


Student  Modeling 

The  pedagogical  decisions  made  by  the  BC  ITS — 
including  choosing  exercise  scenarios,  selecting 
instructional  presentations,  and  controlling  coaching 
behavior — are  all  based  on  the  system’s  evolving 
understanding  of  the  particular  current  student.  This 
understanding  takes  the  form  of  a  student  model.  This 
model  is  composed  of  three  kinds  of  raw  data,  which  in 
turn  is  processed  to  provide  more  directly  useful 
assessments  of  student  state. 

The  raw  data  collected  on  each  student  includes:  (1) 
background  data  (e.g.  demographic  information  on  the 
student’s  prior  experience);  (2)  exposure  data  (e.g. 
records  of  which  exercises  and  instruction  the  student 
has  seen);  and  (3)  performance  data  (e.g.  how  the 
system  has  evaluated  the  student  in  the  course  of 
exercises  and  assessments).  Background  data  is 
gathered  using  the  system’s  question/answer  facility  to 
host  a  survey  questionnaire,  asking  questions  about 
the  student’s  years  in  the  Army,  current  rank,  specialty, 
history  of  deployments,  etc.  Exposure  data  is 
automatically  tracked  based  on  system  actions  such  as 
displaying  instructional  presentations.  Performance 
data  on  exercises  is  logged  in  association  with  coach 
assessments  and  hints.  When  conditions  are  met  that 
could  lead  to  the  coach  giving  positive  or  negative 
feedback,  the  student’s  performance  is  assessed  as 
appropriate  or  inappropriate.  When  the  student  gets 
hints  from  the  coach,  a  smaller  number  of  performance 
points  are  deducted. 

The  raw  data  is  processed  to  provide  estimates  on  how 
the  student  stands  with  respect  to  system  learning 
objectives.  Background  data  can  be  used  to  establish 
initial  estimates  of  mastery  for  particular  objectives. 
Assessment  data  can  be  used  to  update  those 
estimates  based  on  student  performance  within  the 
system.  The  derived  metrics  in  the  system  include  (a) 
score,  (b)  mastery,  (c)  improvement,  and  (d)  progress. 
A  student’s  score  on  a  particular  objective  is  the 
system’s  current  estimate  on  how  competent  they  are 
expressed  as  a  value  between  0  and  1.  Each  objective 
has  a  threshold  score  above  which  a  student  is 
considered  to  have  mastered  the  objective. 

To  maintain  pedagogical  focus,  the  system  uses  a 
strategy  that  emphasizes  a  small  set  of  active  objectives 
at  any  given  time.  Objectives  move  out  of  the  active  set 
when  they  are  mastered  (though  the  system  may 
continue  to  gather  evidence  about  student  competence 
on  the  objective,  which  can  lead  to  downward  score 
revision  and  reversal  of  an  earlier  mastery  decision). 
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Improvement  is  a  measure  of  how  much  a  student’s 
score  has  increased  (e.g.  based  on  performance  in  the 
most  recent  exercise)  on  some  set  of  objectives  (i.e.  all 
objectives,  the  active  objectives,  or  some  particular 
objective).  Progress  is  a  measure  of  how  many 
objectives  have  recently  become  mastered  (e.g.  during 
performance  on  the  most  recent  exercise). 

In  addition  to  the  above  student-specific  metrics 
associated  with  learning  objectives,  we  assume  that  the 
objectives  are  also  statically  ranked  and  linked  in 
various  ways,  such  as  (1)  prerequisite  relationships,  (2) 
subordinate  relationships,  (3)  other  ordering 
preferences,  and  (4)  a  priori  difficulty  expectations . 

Sample  Adaptations 

The  point  of  defining  the  above  derived  student  model 
metrics  and  learning  objective  dependencies  is  to 
enable  statement  of  a  wide  range  of  adaptive 
pedagogical  control  strategies.  Consider  three  classes 
of  decisions  to  be  made  by  the  system:  (1)  which 
instructional  presentations  to  offer,  (2)  which  exercise 
scenarios  to  offer,  and  (3)  which  coaching  interventions 
to  offer.  All  of  these  (and  their  more  specific  variants) 
can  be  tied  to  particular  learning  objectives.  Our 
framework  allows  for  decisions  to  be  based  on  issues 
such  as: 

•  Is  the  learning  objective  in  question  part  of  the 
current  active  set  of  focal  learning  objectives 
for  this  student? 

•  How  long  has  it  been  in  the  active  set?  How 
many  opportunities  has  the  student  had  to 
demonstrate  mastery?  How  much  instmction 
have  they  had  on  this  point? 

•  How  far  from  mastery  is  the  student  on  this 
learning  objective?  How  far  from  mastery  on 
related  (supporting)  learning  objectives? 

•  Has  the  student  been  showing  improvement 
on  that  learning  objective  in  recent  exercises? 
On  related  learning  objectives? 

•  Has  the  student  been  showing  progress 
overall  (or  on  related  objectives)  during  recent 
exercises? 

There  are  a  number  of  additional  system  decisions 
induced  by  this  framework:  (1)  How  many  learning 
objectives  should  be  in  the  active  set  at  any  given  time 
(should  this  be  fixed  or  adaptive)?  (2)  When  should  an 


objective  leave  the  set  (e.g.  upon  mastery)?  (3)  Which 
new  objective  should  be  added  to  the  set  (e.g.  one 
whose  prerequisites  are  either  all  mastered,  or  already  in 
the  active  set)? 

Our  sample  system  has  only  just  begun  to  exercise  this 
machinery.  It  implements  simple  adaptive  strategies  for 
two  classes  of  decision.  (1)  The  system  chooses  initial 
instructional  presentations  based  on  student  answers 
to  an  intake  questionnaire  reflecting  expectations  about 
what  students  with  particular  kinds  of  experience  would 
be  expected  to  know.  (2)  The  system  enables  or 
suppresses  certain  coaching  interventions  for 
individual  learning  objectives  based  on  the  student’s 
mastery  estimates  for  those  objectives. 

A  FRAMEWORK  FOR  EXPERIMENTAL 
EVALUATION  OF  ADAPTATION  STRATEGIES 

One  of  the  technical  goals  of  our  work  is  to  establish  a 
framework  in  which  a  wide  range  of  adaptive  training 
strategies  can  be  implemented.  A  corresponding 
training  research  goal  is  to  establish  tools  that  support 
experimentation  with  adaptation  strategies,  enabling  us 
to  validate  their  utility,  and  characterize  the  conditions 
under  which  they  are  most  applicable.  Likewise,  we 
want  tools  to  help  us  collect  the  kinds  of  data  on  which 
curricular  design  and  adaptation  decisions  must 
ultimately  be  based. 

In  addition  to  putting  the  above-described  technical 
framework  in  place,  we  have  made  some  initial  progress 
on  data  collection  in  the  context  of  our  work  on  BC 
training.  In  June  2010  we  took  a  version  of  the  BC  ITS 
to  Ft.  Knox  and  used  its  integrated  survey  capabilities 
to  gather  two  kinds  of  data  from  a  group  of  nineteen 
junior  officers  and  senior  non-commissioned  officers 
intended  to  represent  the  range  of  potential  BC  trainees. 
We  gathered  background  information  on  these 
subjects,  including  indicators  of  seniority,  rank,  branch 
affiliation,  past  deployments,  roles,  and  relevant 
experiences.  We  also  had  them  take  a  quiz  on 
knowledge  deemed  relevant  to  the  BC  role  (e.g., 
Leibrecht,  et  al.,  2009;  Wampler,  Centric,  Salter,  1998). 
Our  intent  is  to  seek  correlations  between  student 
background  and  existing  levels  of  mastery  for  relevant 
material.  This  is  a  necessary  starting  point  for  any 
adaptive  training  system 

We  envision  follow-on  experiments  to  investigate  the 
efficacy  of  particular  adaptation  strategies  expressible 
within  the  envelope  of  student/objective  conditions 
and  system  pedagogical  decisions  outlined  earlier.  A 
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useful  technical  step  towards  that  end  will  be  to  expose 
the  underlying  conditions  and  decision  as  tests  and 
actions  in  a  scripting  rule  language.  This  will  allow 
experiments  to  explore  a  wide  range  of  strategies 
without  requiring  extensive  programmer  involvement. 

In  support  of  such  experimentation  we  have  already 
begun  development  of  an  observer/controller  (O/C)  tool 
that  connects  to  a  running  simulation  over  a  network. 
The  tool  includes  a  display  for  visualizing  simulation 
activity  as  a  set  of  aligned  timelines  that  can  partition 
and  abstract  complex  multi-agent  interactions.  It  also 
supports  injection  of  some  events  at  the  O/C’s 
discretion.  In  the  future,  it  should  support  definition 
and  selection  of  pedagogical  strategy  rules. 

RELATED  WORK 

Other  researchers  have  developed  systems  that  overlap 
in  various  ways  with  the  approach  described  in  this 
paper.  Scenario-based  training  was  developed  at 
Northwestern  University’s  Institute  for  the  Learning 
Sciences  (ILS)  in  the  1990’s  (Schank,  et  al,  1993).  The 
focus  of  that  research  was  on  identifying  a  family  of 
distinct  scenario  formats  that  would  enable 
development  of  reusable  tools  embedding  the  core  logic 
for  alternate  “scenario  architectures.”  The  underlying 
simulations  often  had  severely  restricted  envelopes  of 
validity,  in  part  because  the  systems  emphasized 
getting  students  to  fail  early  and  often  as  a  way  of 
motivating  learning;  there  was  less  emphasis  on 
confronting  them  with  evolving  consequences  and 
more  on  motivating  them  to  seek  immediate  lessons. 
The  issue  of  being  able  to  provide  a  large  number  of 
individually  adapted  simulation  scenarios  rarely  arose. 

More  recently  and  famously  within  the  military 
community,  work  on  tactical  language  training 
(Johnson,  2007)  shares  many  features  with  the  current 
effort  (and  earlier  ILS  work).  We  are  also  interested  in  a 
military  application  of  scenario-based  automated 
training  that  makes  heavy  use  of  natural  language  input 
and  output.  These  systems  are  responsive  to  varied 
student  behaviors  within  a  scenario.  However  they  pay 
little  attention  to  adaptation  of  scenario  learning 
objectives  and  hence  content  within  and  across 
scenarios.  Natural  language  input  and  output  for 
simulation-based  training  has  also  been  used  with  a 
focus  on  interactive  after-action  review  (Peters  et  al. 
2004). 

Finally,  we  note  that  the  tradition  of  “tactical  decision 
games”  (TDGs,  e.g.  Schmitt,  1994;  Shadrick,  Lussier,  & 


Fultz,  2007)  offers  a  related,  but  somewhat  distinct 
alternative  to  the  kind  of  simulation  we  have  been 
advocating.  TDGs  confront  a  student  with  a  relatively 
complex  scenario  and  demand  some  kind  of  decision. 
Since  they  are  often  paper-based,  they  do  not  provide 
an  evolving  simulation,  and  may,  in  fact,  leave  many  of 
the  situation  details  to  the  student’s  imagination. 
Much  of  the  learning  comes  from  discussion 
(collocated,  synchronous,  and  verbal,  or  distributed, 
asynchronous,  and  written)  where  the  rationale  and 
possible  outcomes  of  alternate  student  solutions  are 
pursued,  ideally  with  expert  commentary. 

CONCLUSION 

Providing  automated  adaptive  training  is  highly 
desirable,  yet  remains  challenging  for  cognitive  skills 
such  as  situation  assessment  and  decision  making.  We 
describe  the  difficulties  that  arise  in  such  applications, 
focusing  on  the  consequences  of  the  need  to 
simultaneously  address  interrelated  constellations  of 
learning  objectives  in  rich  contexts  through  scenarios 
and  simulation.  We  characterize  several  approaches  to 
adapting  instruction  in  such  applications,  and  describe 
a  particular  application,  including  tools  for  gathering 
data  required  to  support  appropriate  adaptation. 

ACKNOWLEDGEMENTS 

Thanks  to  the  many  U.S.  Army  warfighters  who  have 
participated  in  data  collection  sessions  associated  with 
this  project.  This  work  was  supported  by  the  U.S. 
Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  under  contract  number  W91WAW-09- 
C-0013.  The  views,  opinions,  and/or  findings  contained 
in  this  paper  are  those  of  the  authors  and  should  not  be 
construed  as  an  official  Department  of  the  Army 
position,  policy,  or  decision. 

REFERENCES 

Bloom,  B.S.  (1984).  The  2  sigma  problem;  The  search  for 
methods  of  group  instruction  as  effective  as  one-to- 
one  tutoring.  Educational  Researcher ,  13, 4-16. 

de  Oliveira,  M.F.  (1995).  WHAT  NOW,  BATTLE 
CAPTAIN?  The  Who,  What  and  How  of  the  Job  on 
Nobody's  Books,  but  Found  in  Every  Unit's  TOC. 
Combat  Training  Center  (CTC)  Quarterly  Bulletin, 
2d  Qtr  FY  95,  Center  for  Army  Lessons  Learned 


2010  Paper  No  10345  Page  10  of  11 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2010 


(CALL)  Available  at 

http://www.globalsecurity.org/military/library/report/ 
call/  call\_2qfy95\_ctcchapl.htm. 

Duffy,  T.  M.,  and  Jonassen,  D.H.  (1992).  Constructivism 
and  The  Technology  of  instmction:  A  conversation, 
Lawrence  Erlbaum  Assoc.  Inc.  New  Jersey. 

Leibrecht,  B.  C.,  Tucker,  J.  S.,  Haverty,  R.  B., 
Balankenbeckler,  P.  N.,  and  Green,  D.  C.  (2009). 
Metrics  for  assessing  cognitive  skills  in  the  maneuver 
captains  career  course.  ARI  Research  Product  2009- 
04.  Available  at  http://www.dtic.mil/cgi- 
bin/GetTRDoc?AD=ADA495761&Location=U2&doc 
=GetTRDoc.pdf 

Johnson,  W.L.  (2007).  Serious  Use  of  a  Serious  Game  for 
Language  Learning.  In  proceedings  of  the  2007 
Conference  on  Artificial  Intelligence  in  Education. 
67-74. 

Peters,  S.,  Bratt,  E.  O.,  Clark,  B.,  Pon-Barry,  H.,  and 
Schultz,  K.  (2004).  Intelligent  Systems  for  Training 
Damage  Control  Assistants.  In  Proceedings  of 
I/ITSEC  2004.  Orlando  FL,  USA. 

Savery,  J.  R,  &  Duffy,  T.M.  (1995).  Problem-based 
learning:  An  instructional  model  and  its 

constmctivist  framework.  In  B.  Wilson  (Ed.), 
Constmctivist  Learning  Environments:  Case  Studies 
in  Instructional  Design  (pp.  135-150).  Englewood 
Cliffs,  NJ:  Educational  Technology  Publications. 


Schank,  R.C.,  Fano,  A.,  Bell,  B„  Jona,  M.  (1993).  The 
Design  of  Goal-Based  Scenarios.  Contributors. 
Journal  of  the  Learning  Sciences,  3(4):  305-345. 

Schmitt,  J.F.  (1994).  Mastering  Tactics:  A  Tactical 
Decision  Game  Workbook.  Marine  Corps 
Association.  Quantico,  VA. 

Shadrick,  S.  B„  Lussier,  J.  W„  &  Fultz,  C.  (2007). 
Accelerating  the  development  of  adaptive 
performance:  Validating  the  Think  Like  a  Commander 
training.  ARI  Research  Report  1868.  Arlington,  VA: 
U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences.  Available  at  http://www.dtic.mil/cgi- 
bin/GetTRDoc?AD=ADA507445&Location=U2&doc 
=GetTRDoc.pdf 

Vandergriff,  D.  E.  (2006).  Raising  the  Bar:  Creating  and 
Nurturing  Adaptability  to  Deal  with  the  Changing 
Face  of  War.  Washington,  DC:  Center  for  Defense 
Information,  November  2006. 

Wampler,  RL.  Centric,  J„  Salter,  M.S.  (1998).  The 
Brigade  Battle  Captain  -  A  Prototype  Training 
Product.  ARI  Research  Product  98-36.  Available  at 
http://handle.dtic.nil/ 100.2/ ADA347093. 


2010  Paper  No  10345  Page  11  of  11 


